DETECTING QUASICONVEXITY: ALGORITHMIC ASPECTS 



Ilya Kapovich 

Abstract. The main result of this paper states that for any group G with an automatic structure 
L with unique representatives one can construct a uniform partial algorithm which detects L-rational 
subgroups and gives their prcimagcs in L. This provides a practical, not just theoretical, procedure 
for solving the occurrence problem for such subgroups. 



1. Generalized word problem and rational structures on groups 

The goal of this paper is to highlight connections between the theory of automatic groups 
and the generalized word problem and to demonstrate certain additional advantages of the class 
of automatic groups over the class of combable groups. We assume that the reader is familiar 
with the theory of automatic groups, regular languages and combable groups. Although some of 
the important definitions will be given, the reader is referred to [ECHLPT] for further details. 
A good overview of the theory of automatic groups can also be found in [BGSS]. We take for 
granted some basic facts about word hyperbolic groups and their connections with the theory 
of automatic groups. Here our main references are [Gr], [ABCFLMSS] , [ECHLPT], [BGSS] and 
[GS]. An important discussion about combable groups can also be found in [A], [AB] and [N]. 
The author is grateful to the referee for greatly simplifying the proof of Proposition 1 and to 
Gilbert Baumslag for his help in writing this paper. 

Recall that if G is a recursively presented finitely generated group given by a presentation 

G =< Xi, ..,x n \ri, -,r m , .. > (1) 

and H is a subgroup of G then we say that G has solvable generalized word problem with respect 
to H if there is an algorithm which, for any word w in the generators xi, ..,x n , decides whether 
or not w represents an element of H. 

Equivalently, G has solvable generalized word problem with respect to H if the set <^> _1 (-ff) 
is a recursive subset of the free group F{x\, .., x n ) where <j>: F(xi, .., x n ) — > G is the natural 
epimorphism associated with the presentation (1). It is not hard to see that this definition does 
not depend on the choice of a presentation of G with a finite number of generators. 

If G has solvable generalized word problem with respect to the trivial subgroup H = 1 then 
G is said to have solvable word problem . 

The concept of generalized word problem goes back to the work of W.Magnus [M] where he 
proved that if M is a subgroup of a one-relator group G =< X\, ..,x n \R = 1 >, generated by 
any subset of the generating set, then G has solvable generalized word problem with respect to 
M. We should stress that the mere knowledge that G has solvable generalized word problem 
with respect to H does not yet give one an effective procedure for determining whether or not a 
particular word in the generators of G represents an element of H. That is, for practical purposes 
it is not enough to know that the required algorithm exists, it is necessary to be able to find it. 
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Definition. Let G be a finitely generated group. A rational structure on G is a pair (L,A), 
where A is a finite generating set for G, closed under taking inverses, and L is a regular language 
over the alphabet A with ir(L) — G. (Here n: A* — > G is the natural monoid homomorphism 
from the free monoid A* on A to the group G.) We say that (L, A) is a rational structure with 
uniqueness if for any element g G G there is a unique element w € L such that 7r(u>) = g. 

To distinguish the monoid multiplication in A* from the group multiplication in G we will use 
W1W2 for the former and g\ ■ gi for the latter. We will also, when no confusion is possible, denote 
the image tt(w) in G of a word w G A* by w. 

The notion of a rational structure on a group was first closely investigated by R.Gilman in 
[Gi4] who used the term "rational cross-sections" for rational structures with uniqueness. A lot 
of important facts about groups with rational structures, which later were incorporated into the 
theory of automatic groups, can be found in the early works of R.Gilman [Gil],[Gi2],[Gi3] and 
[Gi4]. 

One should think of a rational structure [L, A) for G as of a regular collection of normal forms 
for the elements of G. This point of view is propagated in particular in the work of H. Short [Sho] 
where he investigates different types of rational structures on finitely generated groups. 

Lemma 1. Suppose G =< x\, ..,x n \ri, ..,r m , .. > is a recursively presented group which admits 
a rational structure with uniqueness (L,A) where A = {xf 1 , ...x^ 1 }. Then G has solvable word 
problem. 

Proof. There is a unique word w* G L representing the identity element of G. 

Since G is recursively presented, the normal closure N of the set {n, .., r m , ..} in the free group 
F = F{x\, ..,x n ) is recursively enumerable. Suppose now v is a freely reduced word over x\ 1 ..,x n . 
The language L is regular and so it can be recursively enumerated. Thus one can also recursively 
enumerate the set N x L. We now inspect each pair (n, w) in N x L and check if n = v^ 1 ■ w 
in F. There is a unique w G L such that w = v in G that is v^ 1 ■ w G N . When we find a pair 
(n, w) with n = v^ 1 ■ w in F, we conclude that TJ = 1 in G if w = w* and that v ^ 1 otherwise. 
This completes the proof. 

Corollary 2. (see [ECHLPTJ) If G is an asynchronously automatic group (in the sense of 
[ECHLPT]) then G has solvable word problem. 

Proof. Indeed, by Theorem 7.3.2 and Theorem 7.3.4 of [ECHLPT], the group G is finitely pre- 
sentable and for any finite generating set it possesses an asynchronous automatic structure with 
uniqueness. Thus the statement follows from Lemma 1. 

The following definition, due to S.Gersten and H. Short [GS], is of importance here. 

Definition. Let (L, A) be a rational structure on G and let if be a subgroup of G. 

(a) The subgroup H is said to be L-rational if its full preimage in L, Lh = £H n^ 1 (H), is a 
regular language. 

(b) The subgroup H is said to be L- quasiconvex if there is a constant K > such that for 
any w G L with w G H and for any initial segment wt of w there is a word Ut G A* of 
length at most K such that Wtu t G H. 

An important observation made by S.Gersten and H. Short in [GS] asserts that, adopting the 
notation above, H is L-quasiconvex if and only if H is L-rational. It is shown in the same paper 
that an L-quasiconvex subgroup H is always finitely generated. 

Lemma 2. Let (L, A) be a rational structure with uniqueness for a finitely generated recursively 
presentable group G and let H be an L-rational subgroup ofG. Then the generalized word problem 
for G with respect to H is solvable. 

Proof. Indeed, if v G A* then as the proof of Lemma 1 shows, one can algorithmically find the 
normal form w G L of v. Since 7r~ 1 (iJ) C\ L = L H is regular and is accepted by some finite state 
automaton M H , it remains only to check whether M H accepts w or not. 
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One may think that Lemma 2 completes the discussion about the generalized word problem 
and rational structures on groups. However, as we pointed out earlier, the question of effectively 
finding the desired algorithm requires some further investigation. We will concentrate on the 
following questions. 

Question 1 Suppose a group G =< Xi, .., x n \ri, r m > is given by a finite presentation. 
Suppose further that a rational structure with uniqueness (L,A), where A — {xf 1 , .., x^ 1 }, is 
given explicitly by a finite state automaton M recognizing L. 

Is there a uniform (on H) procedure for finding the language Lh = n~ 1 (H)C\L if the subgroup 
H of G, given by a finite set of generators, is known to be L-rational? 

Question 2 Suppose G =< x\, .., x n \r\, r m > is given by a finite presentation and that 
H = gp(vT, vl) is given by a generating set v\,..,v t . Suppose that we know that there exists a 
rational structure with uniqueness for G such that H is rational. 

Is there a uniform (on H) procedure for finding the automata M and Mh such that (L = 
L{M), A) is a rational structure with uniqueness for G and L(Mh) = Lh = 7r _1 (_ff) n LI 

We will see that the answers to both these questions are positive if we restrict ourselves to the 
class of automatic structures. 

2. Detecting rational subgroups of automatic groups 

Proposition 1. Suppose G =< X\, .., x n \r\, .., r m > is a group given by a finite presentation. 
Suppose (L = L{M),A — {x 1 1 , .., x^ 1 }, M-, M Xl , M x -i ) is an automatic structure with 
uniqueness for G. 

Then there is a uniform (on H) partial algorithm which, for a subgroup H — gp(vT, .., W) given 
by a finite generating set v\,..,Vt, will 

(a) eventually stop and produce the automaton Mh, recognizing 

Lh = 7r _1 (_ff) n L, when H is L-rational 

(b) run forever when H is not L-rational. 

Proof. 

Recall that a Schreier diagram of H in G with respect to the generating set A — {xf 1 , .., x^ 1 } 
is the labeled oriented graph T(G, H, A) whose vertices are cosets Hg, g € G and which has an 
oriented edge {Hg, Hga) labeled by a for each vertex Hg and every a e A. Thus a word in the 
generators A represents an element of H if and only if it is a label of a cycle starting at the vertex 
H = H ■ 1 of T(G, H, A). It follows from the definition of L-quasiconvexity that the subgroup H 
is i-quasiconvex if and only if there is K > such that any word in L representing an element 
of H is a label of a cycle in Y{G, H, A) starting from H and contained in the ball of radius K 
around H in T(G,H, A). 

We will now describe the required algorithm. We may assume that the unit element 1 G G is 
represented by the empty word e € L. If not, we find the word <G L such that w = 1. Then 
L' = (L — w* ) U e is an automatic language with uniqueness for 67. We construct the automaton 
recognizing V and modify the comparison automata M = , M Xl , .., M x -i accordingly. 

Step First, for each word Vi, i = 1, .., n construct the automaton M Vi which accepts all pairs 
(w, u) E L x L such that Wvl = u. Do the same for the inverse of Uj. 

Step 1 Use the Todd-Coxeter method (see ch. 5 of [Si]) to enumerate right cosets of H in 
G. The method produces a sequence of finite labeled graphs Xi with labels from A. These 
Xi approximate bigger and bigger balls in the Schrier diagram T(G,H,A). Each Xi has a 
distinguished basepoint standing for H and the label of any cycle in Xi, starting at the basepoint, 
represents an element of H. 

Each Xi can be turned into a finite state automaton by taking the basepoint to be its initial 
and terminal state. Thus a word over A is accepted by Xi if and only if this word is the label of a 
cycle in Xi starting at the basepoint. Therefore any word accepted by Xi represents an element 
of H. Moreover, for any integer k > the balls of radius k in the graphs Xi eventually stabilize 
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(as i tends to infinity) and map isomorphically (preserving the labels and the basepoints) to the 
ball of radius k in T(G, H, A). Therefore if H is L-quasiconvex, then for i large enough every 
word from L representing an element of H will be accepted by Xi. 

Step 2 Compute the automaton accepting L i7 the intersection of L and the language accepted 
by Xi. For each generator Vj of H check if Li is "stable" under right multiplication by Vj. That 
is, compute the intersection Li x LiOL(M Vj ) and its projection Ni on the first coordinate. Then 
check if Ni — Li. Do the same for vj 1 . If the answers are yes for each j = l,..,t then stop the 
procedure. The output of the algorithm is the language N — Li. If the answer is no, go to Step 
1 and increase i by 1. 

We claim that if the algorithm stops and produces the language N then H is i-rational and 
N = Lh. Indeed, by construction every word from N lies in L and represents an element of H. 
Also, the empty word e belongs to N. It follows now from the description of Step 2 that for any 
k = 1, ..t 

1 e N, Nn C N and Nv^ 1 C N. 

Thus N = H and, since L is an automatic language with uniqueness, N = Lh- 

On the other hand, if H is L-rational, it is L-quasiconvex and, as it was observed in the 
description of Step 1, for some i every word from L representing an element of H will be accepted 
by Xi. It is clear that for such i we have Li = Lh and the language Li is stable under right 
multiplication by Vj and vj , j = l,..,t. Thus the algorithm stops and produces the language 
A /.,/. 

This completes the proof of Proposition 1. 

Proposition 1 implies using the notations above, that the set of L-rational subgroups of G is 
recursively enumerable. 

Proposition 2. Suppose G =< x\, ..,x n \ri, ..,r m > is a group given by its finite presentation. 
There is a uniform partial algorithm which, for a subgroup H = gp(vi 7 .., Vt), given by a finite 
generating set v\, ..,vt, will 

(a) eventually stop and produce an automatic structure with uniqueness 

{L = L(M),A,M = ,M Xl ,..,M Xn -i) 

for G and a finite state automaton Mh such that L(Mh) = k~ \H) H L when H is 
rational for some automatic structure on G; 

(b) run forever if there is no automatic structure for G such that H is rational with respect 
to it. 

Proof. 

The statement easily follows from Proposition 1 and Theorem 5.2.4 of [ECHLPT] which pro- 
vides a uniform partial algorithm for detecting automatic groups and building automatic struc- 
tures on them. 

Most results concerning automatic groups have counterparts for combable groups. The only 
exceptions known to us are the theorem of R.Gilman ([Gi4]) which asserts that infinite groups 
with rational structures with uniqueness have elements of infinite order, the theorem of Epstein 
and Holt [ECHLPT] which states that nilpotent groups are not asynchronously automatic and 
Theorem 5.2.4 of [ECHLPT] which provides a uniform partial algorithm for detecting automatic 
groups. Proposition 1 and Proposition 2, which also have no analogs for asynchronously automatic 
and combable groups, seem to be interesting additions to this list. The author is grateful to the 
referee for greatly simplifying the proof of Proposition 1 . 
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3. Hyperbolic Groups: an alternative approach 

Proposition 1 provides a general algorithm for detecting quasiconvex subgroups of automatic 
groups. There is an alternative approach for detecting quasiconvex subgroups of word hyperbolic 
groups. This approach was suggested to the author by P.Papasoglu and it is indeed very much 
in the spirit of Papasoglu's paper [P] . 

Recall that word hyperbolic groups are ShortLex-autouiatic for any finite generating set (see 
[ECHLPT], Theorem 3.4.5). 

A metric space (X, d) is termed geodesic if any two points can be joined by a path whose length 
is equal to the distance between these points. Such a path, when parametrized by arc-length, is 
also called a geodesic. A naturally parametrized path is called X-quasigeodesic if for any points x 
and y on it 

s < Xd(x, y) + X 

where s is the length of the segment of this path between x and y. A naturally parametrized path 
is called pL-local X-quasigeodesic if any segment of this path of length at most /i is A-quasigeodesic. 
If G is a group and A is a finite generating set of G, we denote the Cayley graph of G with respect 
to A by r(G, A). The word metric on T(G, A) is denoted dA- 

We collect some useful facts about word hyperbolic groups in the following statement. 

Proposition 3. Let G be a word hyperbolic group. 

(i) Let H be a subgroup of G. Then the following conditions are equivalent. 

(1) For some (and so for any) finite generating set A of G and some (and so for any) 
automatic structure (L, A) on G the subgroup H is L-rational. 

(2) H is finitely generated and for some (and so for any) finite generating set B of H and 
for some (and so for any) finite generating set A of G the inclusion (H, d B ) C (G, d A ) is 
a quasiisometry, that is there is some C > such that 

(l/C)d B (h, 1) - C < d A {h, 1) < Cd B (h, 1) + C 

for each h G H. 

We call C a distortion constant for H. 

(3) For some (and so for any) finite generating set A of G the subgroup H is a quasiconvex 
subset of (T(G,A),dA), that is there is an e > (called the quasiconvexity constant for 
H) such that for each h £ H and for any dA-geodesic path w from 1 to h w is contained 
in the e-neigborhood of H . 

(ii) If A is a finite generating set for G and S > is such that the geodesic triangles in 
(r(G, A),cIa) are 5-thin and A > then any lOOOXS-local X-quasigeodesic is global 2X-quasigeodesic. 

Proof. Part (i) of Proposition 3 easily follows from elementary properties of quasigeodesics and 
quasiconvex sets in hyperbolic spaces (see, for example [ABCFLMSS]) and from Theorem 3.4.4 of 
[ECHLPT] which asserts that any automatic structure consists of quasigeodesic words. A careful 
proof is given in [Swa]. Part (ii) is established in [Gr, Lemma 7.2.B]. 

If any of the conditions (l)-(3) of part (i) of Proposition 3 is satisfied, we say that H is 
quasiconvex in G. 

Given a word hyperbolic group G and its finite generating set A, the set L of all (^-geodesic 
words over A is a regular language and is a part of an automatic structure for G. Suppose 
we already have this automatic structure at hand as well as the hyperbolicity constant S of 
(r(G, A), dA) (that is geodesic triangles in (r(G, A), dA) are 5-thin). It follows from the discussion 
about quasiconvex subgroups of word hyperbolic groups in [BGSS] and [GS] that the knowledge of 
the quasiconvexity constant e (and even the distortion parameter C) for a quasiconvex subgroup 
H of G is sufficient for constructing the preimage Lh of H in L. Therefore the following statement 
is equivalent to Proposition 1 for word hyperbolic groups. 
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Proposition 4. Let G be a word hyperbolic group given by a finite presentation G =< x\, .., x n \r\, .., r m > 
and let A = {x^ 1 , .., x^ 1 }. 

Then there is a uniform algorithm which, given a finite set of words vi,..,v t over A, will 

(a) eventually stop and produce the quasiconvexity constant e and the distortion constant C 
of the subgroup H = gp(vi, ..vi) of G if H is quasiconvex in G; 

(b) run forever if H is not quasiconvex in G. 

Proof. 

First wc apply the Knuth-Bendix algorithm described in Section 6 of [ECHLPT] to produce 
the automaton M recognizing the language L = ShortLex(G, A) and the comparison automata 
M=,M Xl ,.., M x - 1 such that 

(L = L(M),A,M = ,M xl ,..,M x -i) 

is an automatic structure with uniqueness for G. This allows us to calculate effectively in G. 
Then we apply the algorithm of Papasoglu [P] or Olshansky [O] to produce the hyperbolicity 
constant 5 of T(G, A). 

Put V = (vi, ..,vt,Vi , ..,v^~ ). Let K be the maximum of the lengths of words v\, ..,Vt- 
Step 1 Start building the balls Ni,N 2 , .., JVf.. of radius i in the Cayley graph T(H, V). 

Step 2 For a current value of i write down all dy-geodesic words of length at most 2i and cal- 
culate minimal A > such that the paths defined by these words in T(G, A) are A-quasigeodesics. 

Step 3 Then for j = i + 1, .., 1000Ki5X check if all dy-geodesic words of length at most 2j 
are still A-quasigeodesics in T(G,A). If for some of these j it is not so, we go back to Step 1 
and increase i by 1. If for each of these j we succeed, then H is quasiconvex in G. Indeed, in 
this case any c?y-geodesic word defines a 1000<5A-local A-quasigeodesic in T(G, A) and therefore 
by part (ii) of Proposition 3 this path is 2A-quasigeodesic in T(G,A). Part (i) of Proposition 3 
implies that H is quasiconvex in G with the distortion constant 2A. 

On the other hand, if H is quasiconvex in G then for some A > all dy-geodesics define 
A-quasigeodesics in T(G, A) and so our algorithm will eventually disclose this fact and stop. 

The promised quasiconvexity constant e for H can be taken to be 1000(5(1 + log 2 C) where C 
is the distortion constant (see [Gr, Proposition 7. 2. A]). 

Example Let k > 2 and 

k 

G =< ai, h, ..,a k , b k \ JJ[oi, h] > 

»=i 

and vi, ..,v t be some words over oi, .., b k defining the subgroup H = gp(W, ■■, W) of G. 

It is well known (see, for example [Swa] ) that the surface group G is word hyperbolic and H is 
quasiconvex in G. Thus the generalized word problem for G with respect to H is solvable. We are 
not so much interested in the reasons for the quasiconvexity and even in finding a presentation 
for H but we still can apply algorithms described in Proposition 1 or Proposition 4 to solve the 
generalized word problem. Moreover, we can practically solve the generation problem for G, that 
is, decide if a given finite collection of elements of G generate G or not. To do this we first 
construct an automatic structure with uniqueness L — ShortLex(G) for G and then find the 
preimage Lh in L of the subgroup iJ, generated by this collection of elements. It remains to 
check if L — Lh is empty or not. 

Remark Notice that for the class of automatic groups solvable generalized word problem does 
not imply quasiconvexity. Let G be as in the example above and be an automorphism of G 
coming from a pseudo-anosov homeomorphism of the surface whose fundamental group is G. 
Take G\ to be the HNN-extension of G along <f> that is 

k 

Gi =< 01,61, ..,a k ,b k ,t\ Y\[a i ,b i ],t~ 1 a 1 t = c/>(cn), ..,t~ 1 b k t = 4>(b k ) > 

i=l 
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Then (see [Th]) G\ is word hyperbolic and there is a short exact sequence 

1— >Z—»1. 

Since G and Z are infinite, by Proposition 3.9 of [ABCFLMSS] G is not quasiconvex in G\. This 
also implies that if L is an automatic language for G\ then G is not i-rational (see [Swa] for 
details). On the other hand the generalized word problem for G\ with respect to G is solvable: 
if v is a word in the generators of G\ then it represents an element of G if and only if the letter 
t occurs in v with the exponent sum zero. 

Notice also that there is an asynchronously automatic structure on G\ for which G is rational 
(see [Sha] for details). At the moment we do not know whether there are finitely generated 
subgroups of asynchronously automatic groups with solvable generalized word problem which are 
not rational for all asynchronously automatic structures on the ambient groups. 

We also do not know if there is an automatic group G and a finitely presentable subgroup H 
such that G has unsolvable generalized word problem with respect to H. E.Rips [Ri] constructed 
an example of a finitely generated (but not finitely presented) subgroup H of a word hyperbolic 
group G with unsolvable generalized word problem. 
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